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1:  introduction 

than  a tlrwriplion  of  a aingle  piece  of  reward*,  lliu  Is  more  in 
e of  a rollrctive  report  on  some  arena  we've  been  ndclrcaalng  in 
*earch  in  stereo  mapping.  We  huvc  l wren  developing  tools  and 
experimenting  with  mulching  strategies  that  wilt  build  to  a rule-based 
stereo  matching  system.  In  particular, .vie  have  been: 

a)  demonstrating  the  design  and  the  utility  of  the  rule-based 
approach  to  surface  inference  from  monocular  Information 
through  the  hand  synthesis  of  matching  strategics; 

h)  developing  tools  to  support  a mapping  interactive  test  facility; 

c)  experimenting  with  the  (Baker  1081]  stereo  mapping  system, 
preparing  to  run  it  on  some  new  imagery.  ( — ..... 

In  a),  we  have  l»een  carrying  out  research  aimed  at  the  analysis  and 
synthesis  of  rides  for  inference  of  three-dimensional  shape  from  single 
images.  We  have  been  addressing  inference  of  matching  rules  mid 
the  use  of  model -based  analysis  both  with  theoretical  analyses  and 
with  hand  anti  automated  aim ly sea  of  specific  matching  strategies;  tin? 
latter  applied  to  both  real  nod  synthesised  imagery  examples.  This 
inference  also  Ims  application  in  constraining  search  for  matches  in 
stereo  mmupomlcitfe. 

Our  work  in  developing  tools  has  centered  on: 

a)  a syc.trm  for  the  hand  construe  Lion  of  edge  descriptions  from 
hard  copy  imagery; 

b)  an  interactive  system  for  determining  the  transform  to  bring 
image  pairs  into  collinear  epipolar  registration. 

Both  or  these  tools  make  extensive  use  of  interactive  graphics,  and 
the  latter  takes  much  advantage  of  previous  stereo  research  from  our 
laboratory  (((lennery  1080]). 

In  c),  we  have  Ism  undertaking  to  apply  the  (Baker  1081]  system  to 
somr  new  imagery.  In  this  we  hope  to  demonstrate  its  elTectivcnesM, 
to  cx|K»sr  its  limitations,  and  to  suggest  both  its  role  in  an  advanced 
mapping  system  and  complementary  research  needed  to  improve  its 
Utility.  Signilirant  restructuring  of  the  system  was  called  for  in  enabling 
it  to  process  this  new  imagery.  Dclnils  of  these  changes  are  described 
in  section  1,  which  deals  with  the  in  niching  process.  Modifications  have 
now  lx*cn  implemented,  enabling  the  system  to: 

• function  on  the  output  of  an  improved  edge  operator 
(Marimont  1082]; 

• use  edge  tzlent  as  one  of  its  parameters  in  socking 
optimized  correspondence; 

• exploit  prepared  transform  information  in  processing 
images  whose  epipolar  lines  arc  not  collinear  with  the 
scanning  axes  or  the  cameras, 

We  will  describe  results  in  these  areas  of  the  research  through  discussion 
of  the  following: 

a)  the  use  of  image  edge  descriptions  produced  using  the  digitiz- 
ing facility  and  from  an  automated  process  (Marimont  1 0R2] 
in  synthesizing  nile*  for  stereo  matching; 

b)  development  of  a system  for  epipolar  registration  of  Image 
pairs; 

c)  modifications  to  the  [Baker  1 1)81]  stereo  system  (results  later).* 


2:  Inference  and  Modelling 

2.1  Proamblc  - the  digitizing  facility 

Our  npprouch  to  rule  development  begin*  witli  hand  synthesized  and 
some  nuloinntieidly  generated  edge  data.  We  have  systems  for  the 
automatic  generation  of  edge  data  ( i. c . [Marimont  1082]).  This  data, 
ext  racted  from  a sufficiently  wide  selection  of  imagery  tyjios,  gives  good 
insight  into  the  current  capabilities  of  automated  processes.  Automated 
processes,  however,  are  not  able  pr  «cnlfy  to  give  as  meaningful  a 
description  of  an  image  as  we  would  like,  nnd  have  not  been  designed 
to  provide  the  uggregraled  abstractions  research  system*  ([!.owe  1982)) 
will  be  soon  supplying.  To  bridge  this  inadequacy,  we  work  with  both 
automatically  generated  data  (the  current  Mtale-of-tlic-art),  nnd  hand 
generated  data  (representative  of  the  next  generation  of  edge  analysis 
processes).  The  hand  generated  data  is  obtained  from  a manually 
operated  digitizing  tablet.  We  have  written  a graphics-based  digitizing 
and  editing  system  to  run  with  a GTCO  tablet  in  producing  these 
image  descriptions. 

Figure  2-1  below  shows  an  Image  pair  of  a building  complex  (referred 
to  as  the  Sacramento  imagery).  Figure  2-2  shows  the  results  of  tablet 
edge  extraction  on  these  images.  Figure  2-2  shows  the  results  of  the 
Marimont  operator  (Marimont  11182]  on  the  image  pair.  Manually 
generated  edge  data  was  produced  using  this  facility  for  the  analysis 
of  rule  synthesis  of  section  2.  It  was  also  used  to  digitize  tin?  building 
data  of  llgures  2-1  for  input  to  tl.c  OTV  inference  process,  as  figure 


2.2  Data  for  Rule  Synthesis 

Wc  have  obtained  extended  edge  data  from  hand  and  automated 
processing  for  use  in  synthesis  of  matching  rules.  Results  from  ear- 
lier work  on  OTV  analysis  (orthogonal  trihedral  vertices)  have  been 
exploited  (Perkins  1908]  for  rule  formation  in  shape  inference  and  in 
constraining  search  for  correspondence.  Wc  have  taken  examples  from 
the  modelling  of  generic  structures  to  produce  ground  and  aerial  views 
of  a building  complex,  and  have  used  this,  as  well  as  other  data,  in  rule 
synthesis. 

2.3  Modelling  and  Vision 

2.3.1  - Modelling,  prediction  and  interpretation 

Of  course,  one  of  the  primary  goals  of  research  in  computer  vision  is 
the  development  of  systems  that  can  recognize  and  locate  objects  in 
images.  In  order  to  identify  such  an  object,  it  is  clearly  necessary  to 
have  some  description  of  its  characteristics  that  can  he  detected  in  an 
image.  A representation  of  an  object  is  the  form  this  description  takes. 

One  approach  to  representation  is  to  provide  the  system  with  three- 
dimensional  models  of  objects.  Rotation  or  these  models  will  nllow 
objects  to  he  observed,  conceptually,  from  differing  viewpoints.  If 
parameters  in  a particular  model  arc  allowed  to  vary  it  is  possible  to 
have  that  single  model  represent  a whole  class  of  objects;  constrain- 
ing the  parameters  functions  to  delimit  sub-classes.  Further  model 
immipiilutiouH,  such  as  partitioning  nnd  projection,  can  be  used  to  nhl 
in  mapping  model  to  Imagery  data.  The  information  contained  in  such 
object  models  may  be  used  to  determine  possible  interpretations  of 
image  features  (e.g.  edges,  ribbons,  corners)  and  to  provide  feedback 
to  predict  the  locations  of  such  features  in  an  image. 

ACRONYM  [Brooks  1981]  is  a three-dimensional  rule-based  modelling/ 
vision  system  developed  here  at  Stanford  that  provides,  among  other 
things,  such  feature  prediction,  model  manipulation,  and  image  inter- 
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Sacramento  Imagery 
I 'inure  2-1 


Manila//)'  K.vlr/icted  /'Mgcs 
Figure  2-2 


Aulonml.irnlly  I’roducal  Edges 

Figure  2-3 


prctalion.  The  rulc-hase  opr  rales  on  tin*  models  and  on  the  sensed 
data  to  accomplish  scene  interpretation.  Such  a rule-based  approach 
has  been  shown  to  hr  an  effective  form  for  constraint  and  search  im- 
plementation, and  allows  easy  modification  and  addition  of  new  rules 
wiLhont  the  nerd  of  altering  the  underlying  code. 

Our  group’s  intention  over  the  next  few  years  is  to  build  a rule-based 
stereo  system  operating  within  ACRONYM  whose  functioning  will  in- 
clude model- based  prediction.  Working  toward  this,  we  have  been  car- 
rying out  experiment*  on  scene  inference  and  model- based  prediction 
that  will  lend  to  a repertoire  of  stereo  matching  rules. 

2.3.2  --  Models  and  stereo  matching 

One  of  the  major  dillicultics  in  determining  stereo  correspondence  is  in 
dealing  with  the  large  nmnlier  of  matches  that  arc  possible.  Solution 
is  generally  found  by  search  through  a large  parameter  space,  where 
possible  correspondences  are  limited  by  geometric  or  photometric  con- 
straints. Search  can  be  reduced  even  more  dramatically  by  endow- 
ing the  matcher  with  hrond  domain  specific  and  ilomnin  independent 
knowledge.  Such  knowledge  can  lx*  rule-bused  arid  model-bused.  Our 
proposition  here  is  that  the  three-dimensional  information  in  object 
models,  along  with  inference  and  prediction  mechanisms,  ran  be  used 
to  interpret  features  in  Imnge  pairs.  These  interpretations  can  then  lx* 
used  as  lllters  to  constrain  the  matching.  We  demonstrate  this  notion 
with  the  example  of  Orthogonal  Trihedral  Vertices,  often  referred  to  as 
culie  corners  or  OTVs.  Other  rules  synthesized  from  analysis  of  Imlli 
manually  extracted  and  automated  edge  processes  follow. 


The  work  on  cube  corners  points  to  additional  usefulness  for  a model- 
based  approach.  OTV  orientation  analysis  (from  matches  across  pairs 
of  views)  yields  almost  complete  solution  for  camera  pnrmambtors; 
constraints  on  sizes  (again,  from  rules  and  models)  could  coinphjte 
the  camera  solution.  But  the  orientation  information  yielded  by  a 
match  of  a pair  of  "crliccs  is  valid  only  if  the  vertex  is  a cube  corner; 
thus  it  is  necessary  to  be  able  to  distinguish  between  vertices  that  are 
cube  corners  and  those  that  arc  not.  If  the  models  contain  sufficient 
information  to  identify  cube  corners,  then  the  problem  of  determining 
cube  corners  independently  of  the  identification  process  is  eliminated. 
In  fact,  both  the  search  Tor  cube  corners  and  the  search  for  identification 
are  likely  to  he  reduced  when  they  arc  combined. 

2.3,3  - OTV  r uij-ba»cd  analysis 

OTV  theory 

In  cultural  scenes,  we  find  a large  number  of  interior  ami  exterior 
corners  of  cubes  typically  when  two  walls  at  right  angles  meet  the  roof 
or  the  lloor.  The  importance  of  utilizing  this  common  structural  ele- 
ment the  Orthogonal  Trihedral  Vertex  (OTV)  has  been  emphasized 
enrlier|l»icbcH  1081 1.  Since  Lhey  provide  a very  tight  constraint  the 
three  edges  are  mutually  orthogonal  In  space  It  is  possible  tocaicumU: 
the  three  dimensional  orientations  given  the  projections  In  the  image, 
'fills  can  he  done  for  both  orthographic  and  perspective  viewing. 

If  the  eye  is  assumed  to  he  focussed  on  the  vortex  of  the  cube  corner, 
perspective  can  he  ignored  and  the  projection  of  a cube  corner  in 
XY  X space  will  simply  be  Its  orthogonal  projection  on  the  XY  plane. 
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Sii|i|HMt'  tlml  some 3 - Blur  lui*  angles  between  tin  rays  u,  h iimi  e umi 
also  llml  llu<  ruyi  me  represented  by  die  mill  vectors  t»j , in,  uj,  Wu  urn 
Interested  in  detcrting  whether  there  urc  llireo  vectors  III  .V  I’  space, 
which  nil-  null  mill)  orthogonal  mill  project,  r«'»|»iTtlv**ly  In  vi,  U),  tij, 

Since  projection  Is  nreompltshed  by  i|rupplii|(  lliu  i component,  miy  3 
such  vectors  u'ii-,1  In'  of  dll'  form  i>i  t X I » , I1)  I X,  j,  mill  uj  I Xa»  wluifo 
i Ib  die  mill  vector  'ii  tin' t direction. 

Requiring  inuliial  orthogonality  Implies  tlml  the  dot  products  of  these 
vectors  in  pairs  In1  u-ru.  From  these  renditions  mill  Billin'  simple 
iiinnipiilulioiiH  we  ran  ralrulaU'  llu-  for imiliut  for 


»i 


/ |ro.  t)  1 Ira.dgra.S,  / |roie)(coil| 

l\r~i sr*r'  v*  “*v “ V — r^T' 


llcttr?  aoiiilbm*  oxUl  if 
n)  ct via,  ro»6,  roue  lire  ull  non-iero  ami 

b)  cither  one  or  three  of  co*«,  coi»b,  coxr  urc  negative,  #o  Mint 
the  quantities  under  the  Hijuarc  root  nign  are  positive. 

These  results  were  first  derived  in  [IVrkins  19C8|, 

Thus  we  have  a way  of  both  eliminating  false  candidates  for  being 
OTVs  and  finding  the  3*1)  orientations  of  valid  OTW  This  algorithm 
has  been  implemented  ami  run  on  data  from  the  digiliting  tablet. 

OTV  with  /from  models 

Our  analysis  begins  on  both  images,  processing  bottom  up  on  the  two 
images  separalely.  As  the  rule  system  nletiti(]es  likely  OTVs  in  images 
(from  its  models),  it  proceeds  In  match  them.  The  system  should 
already  have  a tentative  identification  of  the  buildings  containing  the 
OTVs,  so  there  should  be  relatively  few  possible  matches  nt  this  point. 
Only  OTVs  that  could  In*  the  same  point  on  the  same  object  need  be 
compared.  The  analysis  results  in  depths  of  matched  objects,  for  all 
those  objects  having  OTVs. 

This  requires  that  the  modelling  system  handle  point  elements,  and 
that  it  include  both: 

• inferring  OTVs  from  models  (volumes); 

• accessing  OTVs  stored  explicitly  with  the  models. 

2 A Rule  Synthesis 

J 2.4.  t - Inference  rules 

VVe  continue  with  the  development  of  inference  rules.  This  work  is  a 
logical  extension  of  previous  work  J I I'm  ford  1981,  Lowe  1982)  done  at 
Stanford  in  developing  rules  Tor  inferring  surface  information  from  a 
single  view,  (ictieral  assi  imp  lions  about  illumination,  object  geometry, 
the  imaging  prongs  etc.  have  been  used  to  derive  rules  for  making 
specific  inferences.  For  stereo  vision  Arnold  ami  llitilnril  (Arnold  1080] 
have  developed  roiitlilions  on  correspondence  of  edge  and  surface  inter- 
vals. VVe  divide  our  rules  into  two  categories:  mwiOCAlIncj ijleg,  which 
enable  surface  inference  from  a single  view;  and  Ml rreo  rules.,  which 
facilitate  cross* image  matching. 

2.4.2  - Monocular  and  jdereo  rule#. 

I.  Mo  nor  tdar.ru  lea  Rules  which  have  been  developed  for  in- 

ferences from  monocular  *iews  can  be  utilised  to  provide  a 
partial  Xdimeiihiomd  interpretation  which  directs  search  in 
Llm  second  view.  This  category  includes  the  rule  for  inter- 
pretation of  Ortliogomd  Trihedral  Vertices. 

Another  example  is  the  T-junction  rule  [hinford  MIHI)  which 
stales  that  7»i  nhnener.  of  evidence  io  the.  contrary,  the  stem 
of  n T i*  not  nearer  than  the  top,  i.e.  is  coincident  in  npnet 
or  furlhtr  ntmy\  Application  of  this  rule  gives  a set  of 
nenrer/fnrlher  relations.  A hypothesised  correspondence  of 
edges  which  lends  to  inconsistent  conclusions  from  the  two 
views  enn  Ire  pruned  from  the  search. 


All  1 1 unite  line  wlilrli  In  sirnlglit  iiiiibI  bn  Die  limtgc of  a straight 
h, nice  curve  iiiiIi'bb  the  nirve  ib  planar  anil  Hie  obnorver  I*  torn* 

rlileiilnlly  nils I with  the  piano  of  curvature,  This  cimbloB  U» 

lo  iJIbiiiIbh  correspondences  between  straight  edge*  in  one  view 
nml  rnrvuB  In  the  oilier  view,  lr  two  Inmito  curve*  are  projec- 
tlvoly  romilBlenl  with  parallel,  we  hbbuiiic  they  arc  Image*  of 
curve*  which  lire  pnrnllel  In  *|)iice.  That  linplie*  that  their 
litmgen  in  the  other  view  would  be  pnrnllel  i.e.  parallel*  map 
to  pnrnllel*. 

Ab  Uu-bci  example*  illustrate,  iiiobI  of  the  rulnn  In  (IMnford  1081 , 
I, owe  1082]  nml  other*  developed  by  Mnlik  nml  Hinford  have 
It*  direct  corollaries  stereo  rules  for  checking  the  legality  of  a 
mi, teh.  They  call  even  direct  the  search  process. 

2.  (il .err'd lilies  - these  are  rules  which  have  been  derived  from 

the  slenai  imaging  proresn,  nml  lire  a function  of  the  Imaging 
geometry. 

An  example  rule  in  this  class,  which  lias  long  been  used  for 
finding  stereo  correspondence*,  is  the  cpipolars  rule  - cor- 
responding points  must  lie  on  corresponding  cpipolar  lines. 
These  rules  have  inherently  no  monocular  analogs.  Here  arc  a 
few: 

a)  llorixontnl  planes  in  one  view  get  mapped  to  horiion- 
tu)  planes  in  the  other  view. 

I,)  Use  of  projective  and  qunai-projectivc  invariants.  This 
has  not  been  examined  in  detail.  Duda  mid  llart|l)uda 
1073)  devote  a chapter  to  this  topic  which  lias  not 
really  been  exploited  in  stereo  work. 

r.)  Conditions  on  correspondence  of  edges  and  surface 
inlrrvals[Arnold  I980|. 

d)  S u rf  ii_ce_QcdiJ!til>Ji  Jilkai 

Surfaers  visible  in  one  view  can  be  occluded  in  the 
other  view.  We  are  interested  ill  tile  conditions  when 
this  takes  place.  The  basic  idea  is  that  if  we  cross  a 
surface,  all  obscuration  of  edge  occurs.  A left  surface 
visible  in  a right  view  is  visible  in  the  left  view  un- 
less there  is  obscuration  by  a tall  object.  Similarly  a 
right  surface  visible  in  a left  view  will  be  seen  unless 
obscured  by  a tall  object.  These  surface-obscuration 
rules  can  be  formalised  liy  the  cross-product  rule: 


For  the  hypothesised  edge  match  ct  with  / j and  e, 
with  /a,  we  compute  the  /.-component  of  the  vec- 
tor cross-product  in  the  left  image  pair  ami  the  right 
image  pair.  If  the  *- components  liavr  opposite  signs, 
we  are  seeing  opposite  aides  of  the  surface.  That  im- 
plies that  the  object  is  not  opaque. 

2. Jj. 3 - f/«c  of  inference  rule $ in  a <c»t  analytit 

Our  preliminary  results  indicate  good  potential  for  the  success  of  this 
approach.  On  hand  simulations  with  line  drawings  of  stereo  pairs,  the 
rules  helped  narrow  down  the  choices  considerably. 

Consider  the  imagery  shown  in  figures  2-1  and  2-5.  Figure  2~S  is  the 
right  view  and  2-1  the  left  view.  Vertices  I,  2,  3 arc  orthogonal  trihedral 
vertices.  Using  .lie  formulae  developed  earlier,  we  ran  find  the  3-D 
orientations  of  li  e edge  vectors.  These  can  lie  matched  with  the  3-D 
orientations  of  I',  2',  3'  lo  obtain  a registering  of  these  vertices  when 
combined  with  the  epipolar  constraint.  All  OTV's  in  one  view  need  not 
lie  visible  In  the  oilier  i.t.  f . Of  the  monocular  constraints,  the  other 
major  constraints  which  ran  he  seen  here  are  the  T -junction  rule  and 
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lit**  IMUtllfl*  rule.  In  figure  25  rilin'  !i  in  lirlilml  rili;r  II,  Kilgcs  7 mill 
8 art*  |iarallt*l  ami  so  art*  7'  ami  8’.  A mnlcli  of  8 wll It  II'  would  not 

lir  accepted.  Surface*  ,*>,  ami  ,x',,  arr  both  hiirliniitnl  |>lam'H  (an  rail  hr 
deduced  from  tin*  OTV  analysis)  ami  can  hr  matched.  Surface  St  l«  not 
lioriiontul.  Ilrrc*  of  course,  till*  doe*  not  provide  nny  now  Information. 
Surface  S,  In'ing  a left  fact*  in  a loll  view  in  not  guaranteed  to  ho  vinlhlo 
in  till*  other  view  an  in  fact  it  in.  Tins  crutut-protlucl  rule  could  ho  unod 
to  dlnminn  a match  hrlwren  10  ami  l O'. 


Drurripl  ion  of  heft  Image  of  .Stenn  /’air 
l’'i|;nr«  O-d 


Dr.irriplinn  of  /fight  III luge  of  Stereo  /’air 

Figure  2-5 


S:  Image  Regittration 
3.1  Introduction  to  Epipolar  Geometry 

The  neareli  process  in  automated  stereo  mapping  can  he  greatly 
restricted,  and  roinpiitatioo  l.iinen  significantly  reduced,  if  information 
is  available  relating  the  relative  camera  geometries  of  a stereo  pair. 
Often  this  information  is  available  in  the  reconnaissance  data  (or  at 
least  a rough  approximation  to  it).  Ollier  manual  and  automated 
schemes  have  been  devised  to  provide  the  information  when  it  is  not 
present  with  the  imagery  (see  |llallerl  1 000],  (Gennery  1080]).  This 
camera  geometry  information  nllows  establishing  epipolar  correspon- 
dence of  lines  across  images.  When  this  lias  been  done,  search  in  one 
image  for  match  points  of  a feature  in  the  other  Image  can  be  con- 
strained nlong  a single  vector.  More  generally,  any  features  lying  along 
a particular  vector  in  the  one  image  limy  lie  found  along  a single  vector 
In  the  other  image.  These  image  plane  vectors  are  termed  epipolar 
lines.  Corresponding  vectors  are  termed  corresponding  or  conjugate 
epipolar  lines. 


Ill  this  sei'llon  we  ilctnll  an  algorithm  for  determining  conjugate 
epipolar  lines  In  u set  of  Imagery  fur  which  aiicli  camera  geometry  In- 
formation is  not  explicitly  available.  Mere,  we  rely  upon  mi  operator  to 
select  corresponding  points  in  the  Iwn  imiige*.  The  system  mitonmtl- 
cully  improves  the  resolution  of  the  correspondence  through  Fourier 
Interpolation  over  a match  window  |Gcnnery  1 1180].  The  net.  of  such 
points  is  Inken  liy  an  aiilomnLed  cnniera  solver  In  produce  Ihe  needed 
geometric  liiforiiiiithiii,  This  point  selection  is  done  with  pnn/xoom  cur- 
sor control  on  a graphics  device,  If  the  cniiicrn  infur million  Is  available 
(as,  for  example,  from  recon nalssaiice  data),  tin'll  the  point  matching 
phase  may  lie  omitted  (although  this  provision  Ims  not  I men  enabled 
in  the  current  system).  Ivpinlly,  rough  camera  geometry  information, 
if  avaiialde,  may  lie  used  to  partially  automate  the  point  selection 
phase,  although  again  this  is  nut  implemented  here.  jGcmicry  1980] 
anil  [Moravcc  1980]  have  implemented  totally  automatic  camera  sol- 
vers in  their  stereo  matching  systems.  Our  next  improvement  to  this 
registration  system  will  lie  to  incorporate  l.lie  image  sampling  and  fea- 
ture matching  of  the  (Gennery  1 080]  system,  removing  the  need  for 
manual  point  selection. 

3,2  Gloss  ary  of  Terms 

Given  two  cameras  C i and  ('■•  with  origins  0 , and  0.j  and  focal  planes 
/’ i and  l‘i,  we  call: 

Ikmlk^fJ Lilts;  A family  of  lines  containing  a common  point. 

Kpipolar  ioorilinal.es  of  a point:  The  (lumber  of  (lie  epipolar  line  it 
belongs  to,  and  its  distance  to  a fixed  reference,  like 
the  epipole  if  it  exists. 

Kpipolar  direction:  The  direction  of  all  epipolar  lines  if  they  arc  paral- 
lel. 

Hpipolar  line:  The  intersection  of  an  epipolar  plane  with  a focal  plane, 
Alternate  definition:  the  image  in  one  camera  of  the 
pre-image  of  a point  in  the  other  camera’s  focal  plane. 

Hpipolar  plane:  Any  plane  rnnluin'mg  the  two  camera  centers  0|  and 
~ />,. 

Kpipolar  space:  A space  where  the  coordinates  are  the  epipolar  coor- 
dinates. In  Ibis  space,  a liori/.onl.al  line  is  an  epipolar 
line,  and  the  epipole,  if  it  exists,  is  a wlinle  vertical 
line. 

Kpipolc:  The  intersection,  if  it  exists,  of  all  epipolar  lines  in  a focal 
plane. 

Conjugate  epipolar  lines:  the  intersections  or  all  epipolar  plane  with  the 
two  focal  planes. 


3^  3_  IJrick  g r o_u  nc  1 Theory 

Given  two  stereo  Images  /’,  and  /’.j  (the  content  or  the  focal  planes 
/’ i and  !‘l  or  the  cameras),  and  two  lines  /.,  anil  /,.j  contained  in 
mid  /h,  ill  general  every  point  of  /,,  maps  to  a line  segment  in  l\ , and 
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Him'  lit  mi  inulit'iiliti  rt'liiliuiiitlii)i  lii-Uvit'ii  llm  linn  Hvgimml*  inti|i|iin|f 

til  llillt'U'Ill  (Klllltlt. 

Fumlniimnln)  |irti|ii'rtjrj 

Kvt'l.v  piiiiil  In  /, | will  iiii«|>  In  a llur  m'gimnl  ninliiliinl  In  llinmum>  linn 
l.j  1,  uni  only  If  /,,  mill  f,.j  art’  11  |mlr  nT  iimjugnU,  i'|il|inlnr  tintn. 

Kpipolur  (iitiiiH'tfy: 

Tlio  fuinily  uf  t'pi|inliir  lint'll  in  n fntml  plane  in: 

Kit  Iter 

n)  n st' l nf  puriillt'l  lint's  having  ;i  i-niiinimi  rpipnlitr  tlirt'clion, 


3,5  Akoritlun  U»od 

S.S,  I - Camera  reuhl ration  output 

Until  cinnorii  It  vkiwt'il  iim  ii  rnfm'iillnl  (Oi,x,y,t),  i G {1|J}.  Tlio 
fi’KlHtrilllnll  priimilnrt'  yli'ldn  luimulh,  tltvalion,  pun,  lilt,  •ml  roll  of 
onn  riinii'rn  wllli  rtwpt'tl  lo  llm  other.  Krom  llmru,  wo  compute: 

I)  llm  rntiilion  matrix  /f  liolwoon  llm  Uvo  rcforonlinls: 


ni 

ns 

r2l 

r« 

'as 

r3 1 

fia 

or 

Ii)  a liiimllo  of  Uni's,  llm  inlt  rsiTlion  nf  whirl:  Is  llm  rpipole. 
3.‘1  Requirements  of  tlio  System 
Given  a pair  of  stereo  images,  we  want  lo: 


2)  llm  IrnnsliiLlon  unit  vei  lnr  l,  llm  niinpoimnl*  of  wlileli  are: 


fl  tin  the  base  0 ,xyz,  [ ,i  jin  llm  base  Ojxya 


1)  iilenlify  tlm  kitul  of  epipolnr  geometry  preaenl  in  tile  images; 

2)  explii'ilty  show  the  epipolar  lines  belonging  lo  each  image; 

3)  for  each  image,  compute  llie  parameters  which  relate  the 
original  coordinates  In  the  epipolar  coortlinalrs; 

'1)  conslrnet  llm  image  transforms  in  epipolar  space. 


Top  view  of  a situation  with  oniy  one  epipole 

Figure  3-2 


Top  view  of  II  situation  with  rpipofes 
Figure  3-3 


Prior  to  these  4 steps,  we  will  nrei  I lo  solve  for  the  cameras,  that  Is, 
to  determine  the  fi  parameters  <l««r rihif»K  their  relative  orientation.  A 
prof ccliirr  developed  at  Stanford  [(Jen fiery  ItJHOj  I*  lined  for  this. 


Note  that  Uie  magnitude  of  the  translation  vector  cannot  be  determined 
from  a pair  of  images. 

3.5.2  - Epipolar  ytomtiry  determination 

The  foral  planes  l\  are  planes  paiallel  to  0,xj/,  intersecting  at  z = 
the  focal  distance.  There  is  an  epipole  in  plane  I*,  if  and  only  if  the 
translation  vector  intersects  this  plane,  that  is,  if  and  only  if  its  third 
component  is  not  xcro. 

We  iIiiim  determine  the  case  we  are  working  with: 


if  T '/  0 and  v -/  0,  then,’  sire  two  epipoles:  CASK  l 

if  *7  j/-  0 smd  u — 0,  there  is  one  epipole  CASK  2 

if  ->  = 0 and  v •/  0,  there  is  one  epipole  I'.f.  CASK  3 

if  ^1  as  0 smd  u » 0,  there  sire  no  epipoles:  CASK  4 

Hcnuitkjii 

a)  s-2  0 is  rephiced  in  tile  code  by  \i\  < threshold,  where 
threshold  is  chosen  as  si  function  of  the  arithmetic  precision 
of  the  msichine:  if  we  had  inlinile  precision,  then  we  could 
consider  every  csise  sis  being  case  I.  Here  threshold  ,0001 
wsis  found  to  be  a good  estimate. 

h)  Most  image  psiirs  will  belong  to  the  first  csise,  with  *7  and  u 
of  Lhe  order  of  .1.  The  epipoles  exist,  sire  outside  the  picture 
frame,  and,  for  Lhe  images  worked  with  lo  dale,  tend  to  he  at 
a distance  t>r  about  10  Limes  the  picture  dimension. 

3.5.3  Epipole*  and  epipolar  direction m 

IT  the  epipole  /v,  exists,  it  is  the  extremity  of  Lite  vector  colliuear  to  Ike 
translation  vector,  with  a third  component  equal  to  /,.  If  it  lid's  not, 
then  the  translation  vector  is  the  epipolar  direction.  Hence: 


Cn.su  1: 

M--T'  -r  > 

.* /*/ 1 a/i  v 

'■>?(  i/“  1 V ) 

Cane  2: 

.1  ,nf  1 flh  \ 

M— • — ) 

vM*wO 

Case  3: 

/'2V  u » u ) 

Ch.hu  1: 

V,(n,P) 

3 . 6 Ep ipolar  line  calculation 
3.6.1  - CASE  1:  tmo  cpipoU$ 
a)  theory 

bet  A4i(*tiPii£i)  be  a point  in  !\.  fC\M\  defines  an  epipolar  line  in 
I*  1,  and  lhe  corresponding  delbu*s  the  conjugate  epipolar  line  In 

/a.  The  plane  (/'.’1 , Q\ , 0a,  /••«,  A7|»  5U)  contains  the  translation  vector 
< and  W|M|.  Its  normal  Is  li\M\  X t.  The  normal  of  !\  Is  0*#.  Hence 
tlic  intersection  of  the  two  planes  Is  given  by  the  vector: 
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0>iS  X (h ' t Ki \ X l)  **  {O'iS  • l)W|  M\  - (0a*  • A*  1 M\)t 


•mt  K| Aly  is  tolliucar  tit  this  vvflut  Suppose  that  In  0|*yi,  R\M\i 
(■■  t> l»t  • U).  Ill  trim*  of  ctMliptmeilU  ill  0|*JfJ: 


fl'l  Afy  Is  cull! near  to: 


nf  /V  with  the  plane  (ft1,,  A/|,0»).  Thiii  liny  line  runlalnrd  in  this  plane 
will  intersect  /'«  nt  a imiiil  riiiilnliiiil  In  /-».  In  particular,  conshler  Uie 
parallel  to  U\  driven  through  0».  It  intersects  /'»,  llnni  Aj,  »l  A/j  such 
thill,  In  huso  Otxyr, 


(.  / (.{Ul ‘1 1'JtlL 

ru»i  + ruvi\  >•  'n  n *'ii»i) 

r»i-T|  + r nit i ).  Inwe  M/ 4 

rj|i|  -l  rjtViJ  " r,  ’*  ' 


lv*\  - Xj'A  ^ /xijurn  - Xrj|>  + »|(l-rn  - Xritj\ 

V*r»i  - P3\)  V-»i(er»i  - /irj|)  f yt(nr,*  - lirit)) 

If  we  let  A be  the  matrix: 

/i<r,|  ~ Xrn  el’ll  - Xrjj\ 

Ver-ji  - ;ifj,  pr„  - /iraJ3 

Then  we  can  write  Ay A/y  A A*|  A/,  where  A’,  A/|  in  In  hiuic  0 ixy  anil 
AyA/y  it  in  banc  Otnj. 


h)  algorithm 

I a’ I A,  lie  the  number  of  epi  polar  line*  Unit  we  waul  to  iletcrinine. 
Karli  epipolnr  line  in  uniquely  ileleriiiined  by  the  aiiRle  it  makes  with 
the  x-axis.  Irft  0 ho  this  angle.  (liven  k,  the  cpipolar  line  number, 
0<  fc<  nl-l.  how  ran  we  ilctermine  0?  If  On  anil  0\  are  the  lower 
anil  upper  limit*  helwcn  which  0 is  allowed  to  vary,  and  0a  = , 

then  we  will  chouse  the  middle  of  each  interval:  0 = 0»  + (A  + .5)0 y Hut 
what  arc  0a  and  0 1?  We  have  to  distinguish  lielween  three  cases: 


li)  algorithm 

III  the  smile  way  as  In  raw'  I,  we  ilrllnc  /,|(/’’i , V'i ) where  C|(xi 
rn >“  «iri0y).  Then  t.,  is  ilelined  by  (Afj,  ky),  where  thu  roof- 
illimte*  or  Mj  are 

( . r,ji  1 I riiH/i  r3|j,  -I-  yygiA 

\ * rj  1 x 1 I rjai/i 1 1 r3|X|  rj2i/(  ) 


Cane  2 
Figure  li- 5 

S.6.3  - CASE  3:  one  epipole  Ei 
a)  theory 

Let  an  cpipolar  line  in  P\  be  denned  by  the  translation  vector  t = V'i 
and  by  a point  M 1 we  pick.  In  l\,  Ay  Roes  throuRii  A’y  and  is  coliinear 
to  a vector  A'y  M y , intersection  of  I’-J  witli  tiic  plane  (0y , A’y , Af  1 ).  This 
plane  is  orthogonal  to  ^2 Af  1 X t and  fy  is  orthogonal  to  0yr.  Tlcncc 
the  intersection  is  coliinear  to  0yj  X (0yA/|  X l) 


• The  rpi|H,le  is  in  the  picture  (very  unlikely).  Then  0 
can  vary  ticlworn  0 and  2ir  radians: 

• The  epipole  is  outside  the  image:  there  is  a minimum 
and  maximum  angle  under  which  the  image  is  seen 
from  this  (mini.  If  we  choose  tliese  angles  in  (0, 2s], 
then  most  of  the  time  every  0 0 [0o.0||  will  define  a 
valid  cpipolar  line  in  /’ 1; 

• The  exception  from  above  is  when  the  epipole  is  left 
of  the  image  hut  on  a same  vertical  level:  then  (Do,  0i] 
cannot  be  connected  and  still  included  in  (0, 2*].  In 
thin  case  we  will  choose  the  angle*  in  |^,  |). 

Then  A|  will  he  definrd  by  the  point  and  the  vector  V\[coa0,tin0), 
and  1*1  la  defined  by  fc’j  and  V'i  — • AV\. 

3.6.2  - CASE  2:  one  tpipolt  Hi 

«)  theory 

Olvrn  an  cpipolar  line  jfyf;,  we  already  know  that  the  corresponding 
cpipolar  line  I, j in  J'j  is  coliinear  to  the  vector  I -»  Vj (X, /<).  Ilenco  we 
Just  need  to  find  a (mint  lielonging  to  Aj.  Clearly,  f, % is  the  intersection 


since  0}M\  = 0y0|  + 0|  Af , 

= kt  4-  0|  Af|, 

0yAf,  X t = 0,  Af,  X I, 

and  Cyc  X (0yAf,  X l)  = (0ylt)Mf|  - (0yc  • 0|  Af  ,)l. 
Suppose  0\M]  : (zi,!/ii/i)  in  0|Ty;.  Then  in  0yi|/*: 

0/>.\  /i'\  /rni|  +ri»yi  >C|j/|\ 

1 1 — n 1 0i  Mi  — I y’|  I "•  I ry!i|  + rjyyi  + rjj/i  ] 

V'  \r'i  / vsi»i  + rjim +rjj/i/ 

/v’yA/j  is  thus  rolllnear  to: 

f e*  1 - A«',\  __  f li(ern  - Ar.o)  I-  Vl(‘"n  - Xrjj)  +/,(»,. i - Xrjj)  \ 

VI'V'i  -nr'i/  \*i(eryi  - (Oji)  + m(i'r*»  - prjy)  f /i(erH  - prjjJy 

Hence,  If  we  let  /I  lie  the  same  matrix  ns  In  CASH  I and  OK3  lie  the 
ulfsel  nintrlx: 
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ft-r i)  - Xr»A 
Wr,j  - lira) 


Thin  mv  liiivv;  KtMt  . AOt\lt  i OKI 

Whvff  <Mf|  U ill  how  tf,/y  mill  h’„(Wj  U in  him*  0,jy 

l»)  »|n»fUhm 

N.m,  him  .In  |,irk  ,W,  in  I l,o  f|,»t  |,|,w,.»  Wo  mini  i,  not  ,,r  N,  onnnlly 
•IMWoil  0|.||.,.h,f  liii.il,  mill  it  II|I|MHIH  t'.iiivoiiioiil  u.  pick  |miImU  on  ll.o 
.‘"7  i t,u'  ‘P'l’olnf  •»»«  nro  innrr  liiirii.inlnl,  or  llio  liini|(o  »lrotrlicil 
m livmlil,  Ihoii  MO  will  pith  .V,  iipmlly  Hpnr.-.l  points  oii  Hip  vorllrnl 
Hi.Hnl.ly  hwiilo.l  lo  .-..vor  iho  o.illro  ininKo.  ir  tho  ophioln,  ||„on 
Ml  11.010  y.'tlicnl  or  ll.o  imi.Ko  nioro  Mr.  li  h.il  in  wi.llli,  then  wo  nl.k 
...m  oii  ilio  horiionlnl  .,mv  l,.  i |H.  i|„.  ,,loi„ro  .II,„o„hIo.,h  ii.i.I 

l*..*»)  llio  c|»|iuliir  iliroriion: 

If  < V,L,,  nml  V,  < o,  wo  pick 

IfVi  o-.r  . 

If  '.In  < If, I., , nn.l  V,  > 0,  we  pick 

If  l \l.  < V,l.„  mu)  V',  < 0,  wo  pick  ^ ' 

*•-('•»  lSjl  + /..j(K*U) 

« *|/-j  < V*t. fc,  ami  \\  > 0,  we  pick 

Tlion  wo  prorooil  nri  in.licnlcil  nlmvc: 

/.|(Afi,V'l  = I)  in  iiinlrliiil  with  h'jM}). 


S,0,4  - CASK  ii  no  tpipnlf 
«Q  tlnory 

l.ot  mi  oplpohir  lino  In  l’t  l>o  iluDniil  l.y  tho  tmn»lslion  vector  I *•  K> 
kimI  l.y  n |H)inl  A/|  wo  pick.  In  I j,  /.j  |i»r»  Itiroiint.  llio  Imtgoof  M i, 
llml  in  llio  extremity  of  n vector  rollinonr  to  0\M\  and  wIiom  third 
component  I * ft.  In  I.iiho  0\jyst0\K1\  : (z  i . Vi  • /i  )•  In  b»»c  0fXyi: 


/m*i  + r„y,  +r„/,\  ( ft 

'M'  “ r*'*'  + T«V<  + ) C»W*  1=1 

\TliXi  + r»yi  + fia/i/  \ ^ 


ii)  algorithm 

We  pick  poihlH  | In  the  Maine  innniicr  a*  in  rune*  3.  Then  we  calculate 
Oj  A / 'j  im  indicated  above  mid  we  match  //((/'’ j , ()  with  hi ( /v j , < ) , 

37J-:in p oU r Rc^islralioh  and  TranHlormation 

l*'inure  3-H  nhowa  h ntereo  pair  of  i\  l>mhliti}(  romp  lex.  Kiftiire  3*7  ha* 
th is  pair  HiiperpoHilioiied  with  a net  oT  rorrcttpondiitf’  epipolur  Utica. 
Figure  3-8  nhowa  the  imagery  l.raiiMformcd  Hindi  Unit  cpipohir  linen  arc 
liorizoiil.il  in  the  image,  ami  conjugate  e pi  polar  l in  cm  have  Lite  same 
row  coordinate. 


Tr/w*  formed  fm tilery 
Klpire  3-8 
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4<*  Automaled  Stereo  Mapping 

4.1  Background 

Result*  from  our  laboratory  ovrr  the  past  few  year*  {Quum  1971, 
|l*uimh  1071,  Moiuvcc  I OHO,  (ivmivry  1080,  Arnold  1080,  linker  1081, 
Arnold  1083],  him*  dtommst  ruled  the  poiudblllLiuft  of  both  urea- baaed 
and  fruluro  based  atemi  matching. 

Ar<n  bn*tJ  *ti*i i*o  matching  urn**  windowing  mechanisms  to  Isolate 
pat  la  of  two  tillage*  for  r root- ror  relation.  Featu  rr- Aoifd  stereo  match- 
ing use*  two-dimensional  convolution  operator*  (nml  |K*rhnp*  grouping 
operators)  to  reduce  un  iinngc  to  n depiction  of  It*  loir  unity  bouiul- 
uvie*,  ahieli  can  then  In*  pul  into  correspondence.  Aren-bnned  crosa- 
rorrelatino  lechiihpieH  require  distinctive  texture  within  the*  area  of  cor- 
relation for  successful  operation.  In  general,  it  breaks  track  where  there 
is  no  local  correlation  (lero  signal,  or  where  two  image's  do  not  cor- 
respond, i.«.  occlusions)  or  where  the  correlation  is  ambiguous  (where 
the  signal  Is  repetitive). 

Demand*  of  mapping  in  cultural  sites  nml  in  locales  with  surface  discon- 
tinuity ami  ambiguous  or  nonexistent  texture  make  it  essential  that, 
if  area- bas<*d  analysis  is  to  be  done,  it  he  done  in  conjunction  with 
feat ure- based  analysis.  Kriiturr- based  analysis  provides  a solution  to 
many  of  the  problems  of  correlation.  Principal  among  its  advantages 
is  that  it  operates  on  the  most  disrrirninahle  parts  or  an  image:  places 
that  are  distinctive  in  their  intensity  variation,  and  where  (ocnlhnlion 
i*  greatest.  These  are  typically  the  boundaries  between  objects  or  be- 
tween detail*  on  objects,  or  between  objects  and  their  backgrounds. 
The  important  point  is  that  the  features  being  pul  into  correspondence 
for  depth  estimate*  are  the  Umndaries  »»f  objects:  area-based  analysis 
is  it  its  worst  at  object  Unindaries,  yet  determining  boundaries  can  be 
said  Lo  !h*  the  most  important  part  of  mapping  in  3- space. 

Tin*  [I taker  IflHI]  system  is  the  only  current  system  t hat  mixes  these  two 
matching  moduli*  ie*.  We  have  been  working  at  applying  this  system  to 
M»me  cultural  scenes,  lie  fore  carrying  out  these  analyses  we  wished  to: 

a)  enhance  the  system  with  a capability  lo  work  with  a better 
edge  operator  [Marimonl  1082]; 

b)  enable  it  to  process  images  that  are  not  graced  with  colli  near 
epipolar  geometry  (i.r.  most  images); 

c)  introduce  an  additional  correspondence  measure  edge  extent. 

4.2  Itpipolar  Registration 

To  implement  these  enhancements  required  substantial  redesign  of  the 
systrm,  and  redesign  cycles  with  the  Marirnont  process.  Chosing  use- 
able  data  also  presented  difficulties,  as  the  only  imagery  available  was 
not  of  the  correct  geometry  (sec  below).  The  two  image  pairs  initially 
rhuneh  (the  Sacramento  apartment  complex  and  a section  of  some  im- 
agery of  MofTcll  Field)  proved,  on  closer  examination,  to  require  quite 
complex  transformation,  and  could  not  bo  easily  adjusted  for  epipolar 
processing. 

In  general,  to  bring  imagery  data  into  a properly  transformed  state 
mold  proceed  in  one  of  two  ways: 

• one  could  determine  the  transforms  and  then  modify 
the  imagery,  producing  an  iniAgc  pair  having  col  linear 
epipolar  geometry; 

or 

• one  could  determine  the  transforms,  and  modify  the 
output  of  an  edge  operator  process  that  functions  over 
the  original  imagery. 

The  Utter  in  by  fnr  the  superior  approach,  as  it  avoids  resampling 
the  image.  This  approach  necessitates  incorporating  the  transform 
computation  into  the  stereo  system,  lo  follow  edge  finding  and  precede 
edge  matching. 

The  second  part  of  the  stereo  system's  analysis  is  an  Intensity  correla- 
tion process.  This  operates  along  epipolar  lines  as  well,  and  clearly 
requires  intensity  informnlion  lo  be  accessible  along  epipolar  lines.  One 
solution  to  this  would  be  to  take  the  original  Image  pnir  anti  have  the 
correlator  rotate  and  change  shape,  s'ur,  and  orientation  as  It  move* 
around  the  image;  ibis  is  an  awkward  nml  probably  unnecessary  com* 


plication.  An  alter native  would  he  to  access  the  transformed  images, 
sampled  its  accurately  as  possible,  nml  do  the  correlation  In  the  rectan- 
gular space  dr  lined  by  roll  incur  epipolar  lines,  'fhe  argument  from  edge 
accuracy  indicated  that  transforming  edges  rather  then  resampling  the 
image  was  the  way  to  go;  this  argument  from  intensity  correlation  sug- 
gests that  the  resampled  Image  can  be  useful. 

Another  implementation  detail  supported  this  use  »r  both  transformed 
edges  and  transformed  imagery:  it  was  found  that  the  intensity  in- 
formation available  from  the  Marimonl  process  had  too  small  a basis 
for  useful  correlation,  and  iri  fact,  for  transformed  edges,  hurl  little 
relevance  for  the  matching  (if.  being  measured  not  along  epipolar  lines, 
but  normal  lo  the  edge  direction).  The  transformed  image  bad  to  l>c 
referenced  again  by  the  system  to  obtain  more  significant  intensity  es- 
timates oriented  along  epipolar  lines,  arid  working  with  the  image  in 
epipolar  space  facilitated  this. 

The  philosophy  of  the  stereo  matching  process  here  bad  been  to  use 
edge  analysts  for,  among  other  things,  its  higher  ar»**  — ryt  and  to  use 
intensity  analysis  for  the  continuity  it  provides.  To  i.  onsislonl  with 
this,  we  wanted  lo  have  the  highest  possible  accuracy  for  edges  *.n 
epipolar  spare,  and  if  sacrifice  be  needed  for  simplicity,  to  do  it  wl  ere 
it  least  degraded  the  analysis  - in  the  intensity  correlation.  It  is  clear 
that  transformed  edges  give  higher  accuracy  than  edges  from  trans- 
formed images  (detectability  might  not  change  much,  hut  localisation  is 
significantly  reduced);  and  important  simplifications  could  be  obtained 
for  lillle  loss  by  lining  the  intensity  correlation  over  the  resampled 
image  pair.  This  meant  changes  in  our  plans  for  the  registration  sys- 
tem: it  bad  to  produce  not  just  transform  information,  but  transformed 
images  ;ls  well.  Ilotli  forms  are  made  available  as  output  from  the 
registration  program  described  in  section  3,  and  Lite  enhanced  linker 
system  uses  them  both. 

4.3  The  Marirnont  Edge  Operator 

The  Marirnont  edge  operator  lots  greater  detection  and  reliability  than 
the  original  linker  edge  operator,  and  similar  localisation;  earlier  ex- 
ample of  its  processing  convinced  us  that  its  output  would  improve 
the  quality  of  our  stereo  reconstruction.  Its  ability  to  track  along  scro 
signal  areas  in  following  uoro- crossing  edges  leads  Lo  more  coherent 
image  descriptions.  [Marirnont  1082]  provides  details  of  the  operator's 
functioning.  Roughly,  it  works  by  convolving  an  m X m lateral  inhibi- 
tion function  of  n X n central  window  with  an  image.  Zero  crossings 
in  this  resultant  image  then  indicate  edges,  ami  the  edge  position  is 
determined  by  interpolating  over  the  lateral  inhibition  surface. 

A few  unanticipated  problems  became  apparent  once  work  with  the 
edges  was  begun.  One  point,  noted  above,  was  that  the  intensity 
information  stored  at  an  edge  (its  left  and  right  boundary  values)  had 
quite  small  support  (a  single  pixel).  This  is  in  contrast  with  the  original 
operator  which  interpolated  for  these  values  in  an  area  3 pixels  wide  and 
removed  one  pixel  from  the  determined  edge  position.  Another  problem 
was  that  the  edge  connectivity  produced  by  the  Marirnont  system  can 
be  misleading.  Intensity  significance  was  improved  by  sampling  along 
epipolar  lines  in  the  transformed  images.  The  connectivity  problem 
has  not  been  looked  at  yet.  Good  connectivity  is  inherently  difficult  to 
nehieve  with  tero  crossing  operators.  Refinements  to  the  process  arc 
being  considered. 

■1.4  Edge  Extent 

The  introduction  or  edge  extent  as  a parameter  in  the  dynamic  pro- 
gramming solution  was  an  obvious  fallout  from  using  the  Marirnont 
edges.  lidges  are  output  by  that  process  ns  strings,  with  2- 
connectedness.  The  maximum  and  minimum  of  some  string,  in  trans- 
form space,  is  a measure  of  its  (epipolar)  extent,  I’rior  to  the  rise  of  Ibis 
information  the  only  way  that  global  continuity  entered  the  analysis 
was  through  a consistency  enforcement  relaxation  process  which  en- 
sured that  edges  connected  in  one  view  were  interpreted  as  continuous 
In  3-apacc;  nil  matching  measures  were  quite  local.  With,  the  modified 
approach,  the  correspondence  measure  is  a function  of  (among  other, 
more  statistically  based  parameters)  tire  ratio  of  erlgc  extents,  In  par- 
ticular, tlru  likelihood  of  edge  element  n in  the  left  image  matching  edge 
clement  b In  the  right  Image  depends  on  the  product  of  the  ratios  of 
the  two  upper  exlrnts  (up  from  the  edge  elements)  and  the  two  lower 
rxlcnls  (down  from  the  two  edge  elements). 
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4.5  Testing  on  Imagery 

When  Range  Idling  begun  with  nil  of  the  ubovo  accomplished,  another 
problem  Iwcumi*  apparent:  the  stereo  system,  bound  into  n machine 
arc  hi  tec  In  re  with  a maximum  of  258  K words  of  memory,  and  alwnyii 
tightly  wedged  anyway,  had  grown  with  thorn)  changes  to  the  point  that 
only  small  portions  of  images  could  he  worked  on  lit  once.  Tims  cninc 
to  exist  a windowing  mechanism  within  the  edge  lUiding/londihg  and 
stereo  matching  processes. 

Our  testing  has  been  progressing  on  several  sets  of  imagery:  a syn- 
thetic image  pair  from  Control  Data  Corporation,  an  aerial  scene 
from  the  Ktigineoring  Topographic  Laboratory,  and  a building  scene 
or  Sacramento.  Wc  will  report  on  the  results  of  these  analyses  at  a 
later  date. 

5:  Summary 

A principal  research  interest  of  otir  group  is  in  developing  a fule- 
hnsed  advanced  automated  stereo  mapping  system  to  function  within 
ACRONYM  [Rrooks  1981).  Current  mapping  techniques  ignore  much 
of  the  information  available  from  inference  on  single  views  of  a scene. 
This  information  can  be  useful  for  three-dimensional  surface  interpreta- 
tion, and  also  provides  extra  parameters  for  stereo  matching  (i.c.  sur- 
face orientation,  occlusion  cues).  Our  research  cITort  is  directed  at 
establishing  such  monocular  inference  rules  in  a rut>basc  for  stereo 
mapping. 

In  deriving  these  rules,  we  perform  analysis  of  both  hand  extracted 
ami  automatlrally  produced  edge  descriptions.  A facility  has  been 
developed  for  this  manual  edge  extraction  from  hardcopy  imagery. 
Wc  have  studied  rule  synthesis  for  several  cases,  including  that  of 
orthogonal  trihedral  vertices  - features  that  dominate  cultural  scenes. 
This  research  is  very  promising,  and  has  shown  the  utility  of  the  rule- 
based  approach  to  surface  inference  front  monocular  information. 

Camera  solving  provides  powerful  constraint  on  the  correspondence 
problem  in  stereo  matching.  Wc  have  developed  a facility  for  interac- 
tively registering  images,  determining  the  parameters  for  transforming 
them  (or  their  edge  descriptions)  into  collinear  cpi polar  space,  and  per- 
forming the  actual  image  transformation.  This  determination  is  crucial 
to  a mapping  process.  Incorporating  an  automated  module  to  provide 
data  for  the  camera  solving  is  a very  important  next  step. 

Wc  have  expcrii  irnlcd  with  an  existing  stereo  mapping  process,  en- 
hancing its  flexibility  with  respect  to  image  format  and  with  respect 
to  edge  operator  format,  and  have  been  preparing  example  outputs  of 
its  processing  on  new  imagery.  Our  intent  with  this  cITort  has  been 
to  show  the  capabilities  or  a local  matching  process  and  to  assess  its 
applicability  to  the  planned  rule-based  system. 
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